1 Domain problem formulation

What is the real-world question? This could be hypothesis-driven or discovery-based.

Some advice

Why is this question interesting and important? What are the implications of better understanding this data?

Briefly describe any background information necessary to understand this problem.

Some advice

Briefly describe how this question can be answered in the context of a model or analysis.

Outline the rest of the report/analysis.

2 Data

What is the data under investigation? Provide a brief overview/description of the data.

Describe how your data connects to the domain problem.

2.1 Data Collection

How was the data collected or generated (including details on the experimental design)? Be as transparent as possible so that conclusions made from this data are not misinterpreted down the road.

Describe any limitations when using the data to answer the domain problem of interest.

Where is the data stored, and how can it be accessed by others (if applicable)?

2.2 Data Splitting

TODO: add advice for possible data splits, AK getting nice figure together

Decide on the proportion of data in each split.

Decide on the “how” to split the data (e.g., random sampling, stratified sampling, etc.), and explain why this is a reasonable way to split the data.

Split the data into a training, validation, and test set.

data_split <- basicDataSplit(X = X, y = y, 
                             train_prop = params$train_prop, 
                             valid_prop = params$valid_prop, 
                             test_prop = params$test_prop,
                             stratified_by = y)
Xtrain <- data_split$X$train
Xvalid <- data_split$X$validate
Xtest <- data_split$X$test
ytrain <- data_split$y$train
yvalid <- data_split$y$validate
ytest <- data_split$y$test

Provide summary statistics and/or figures of the three data sets to illustrate how similar (or different) they are.

2.2.1 Data Splitting Overview

X Data Split

plot_X_datasplit(Xtrain, Xvalid, Xtest)

Y Data Split

plot_y_datasplit(ytrain, yvalid, ytest)

2.3 Data Cleaning and Preprocessing

What steps were taken to clean the data? More importantly, why was the data cleaned in this way?

Discuss all inconsistencies, problems, oddities in the data (e.g., missing data, errors in data, outliers, etc.).

Record your preprocessing steps in a way such that if someone else were to reproduce your analysis, they could easily replicate and understand your steps.

It can be helpful to include relevant plots that explain/justify the choices that were made when cleaning the data.

If more than one preprocessing pipeline is reasonable, examine the impacts of these alternative preprocessing pipelines on the final data results.

Again, be as transparent as possible. This allows others to make their own educated decisions on how best to preprocess the data.

Xtrain <- log(Xtrain + 1) %>%
  removeConstantCols(verbose = 1) %>%
  removeDuplicateCols(verbose = 1) %>%
  filterByVar(max.p = 1000)
#> Removed 15 features
#> Removed 5 features
Xvalid <- log(Xvalid + 1)[, colnames(Xtrain)]
Xtest <- log(Xtest + 1)[, colnames(Xtrain)]

2.4 Data Exploration

TODO: Add drag and drop feature in shiny version for other images

The main goal of this section is to give the reader a feel for what the data “looks like” at a basic level.

Provide plots that summarize the data and perhaps even plots that convey some smaller findings which ultimately motivate the main findings.

Provide additional plots representing remaining oddities after pre-processing if applicable.

Add summary statistics in accompanying tables (or in figures) for quick comparisons.

2.4.1 Training Data Overview

#> Number of features: 1000
#> Number of training samples: 649
#> Number of NAs in training y: 0
#> Number of NAs in training X: 0
#> Number of columns in training X with NAs: 0
#> Number of constant columns in training X: 0

Summary Tables

# summary of types of features in training X
data_types(Xtrain = Xtrain, ytrain = ytrain)

# summary of features in training X, sorted by type
tab_ls <- data_summary(Xtrain = Xtrain, ytrain = ytrain, digits = 2, sigfig = F)
for (dt_name in names(tab_ls)) {
  simChef:::subchunkify(tab_ls[[dt_name]], i = chunk_idx, 
                        other_args = "results='asis'")
  chunk_idx <- chunk_idx + 1
}

X distribution

# plot X distribution
plot_X_distribution(Xtrain, "density")

Y distribution

# plot y distribution
plot_y_distribution(ytrain, "bar")

Feature Correlation

# correlation heatmap
keep_features <- sort(
  sample(1:ncol(Xtrain), min(ncol(Xtrain), 1000), replace = FALSE)
)
plotCorHeatmap(X = Xtrain[, keep_features], cor_type = "pearson", 
                      clust = TRUE, text_size = 0) +
  ggplot2::labs(x = "Features", y = "Features", fill = "Cor.") +
  simChef::pretty_ggplot_theme(x_text_angle = TRUE)

Feature Pair Plots

# pair plots
keep_features <- sort(
  sample(1:ncol(Xtrain), min(ncol(Xtrain), 6), replace = FALSE)
)
plotPairs(data = Xtrain, columns = keep_features, 
          color = ytrain, color_label = "y")

Marginal Associations

caret::featurePlot(x = Xtrain[, keep_features],
                   y = ytrain,
                   plot = if (is.factor(ytrain)) "box" else "scatter",
                   # strip = strip.custom(par.strip.text = list(cex = .7)),
                   scales = list(x = list(relation = "free"), 
                                 y = list(relation = "free")))

PCA

# dimension reduction plots
plotPCA(X = Xtrain, npcs = 3, color = ytrain, color_label = "y",
        center = T, scale = FALSE)$plot

3 Prediction Modeling

TODO: add advice on which models to select and why

Discuss the prediction methods under consideration, and explain why these methods were chosen.

Discuss the accuracy metrics under consideration, and explain why these metrics were chosen.

Note: there should be multiple methods and metrics under consideration to paint a more holistic picture of the data. At least one method should be a baseline, common approach that may not be optimal for the problem setting, but serves as a helpful comparison.

3.1 Prediction check

Carry out the prediction pipeline, outlined above.

Fit prediction methods on training data.
Evaluate prediction methods on validation data.
Compare results, and filter out poor models.

## Pick models and fitting backend for caret
tr_control <- caret::trainControl(
  method = "cv",
  number = 5,
  classProbs = if (is.factor(ytrain)) TRUE else FALSE,
  summaryFunction = caret::defaultSummary,
  allowParallel = FALSE,
  verboseIter = FALSE
)

response_type <- "raw"
model_list <- list(
  ranger = list(tuneGrid = expand.grid(mtry = seq(sqrt(ncol(Xtrain)), 
                                                  ncol(Xtrain) / 3,
                                                  length.out = 3),
                                       splitrule = "gini",
                                       min.node.size = 1),
                importance = "impurity",
                num.threads = 1),
  xgbTree = list(tuneGrid = expand.grid(nrounds = c(10, 25, 50, 100, 150),
                                        max_depth = c(3, 6),
                                        colsample_bytree = 0.33,
                                        eta = c(0.1, 0.3),
                                        gamma = 0,
                                        min_child_weight = 1,
                                        subsample = 0.6),
                 nthread = 1)
)

start_time <- Sys.time()
model_results <- fitCaret(Xtrain = Xtrain, ytrain = ytrain, 
                          Xtest = Xvalid, ytest = yvalid, 
                          model_list = model_list, tr_control = tr_control, 
                          response_type = response_type)
#> [11:07:23] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:23] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:23] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:23] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:23] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:23] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:23] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:23] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:31] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:31] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:31] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:31] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:31] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:31] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:31] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:31] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:35] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:35] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:35] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:35] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:35] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:35] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:35] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:35] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:40] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:40] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:40] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:40] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:40] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:40] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:40] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:40] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:47] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:47] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:47] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:47] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:47] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:47] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:47] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:47] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:56] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:56] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:56] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:56] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:56] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:56] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:56] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:07:56] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:00] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:00] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:00] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:00] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:00] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:00] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:00] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:00] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:05] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:05] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:05] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:05] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:05] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:05] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:05] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:05] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:12] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:12] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:12] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:12] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:12] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:12] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:12] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:12] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:25] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:25] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:25] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:25] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:25] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:25] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:25] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:25] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:30] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:30] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:30] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:30] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:30] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:30] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:30] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:30] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:37] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:37] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:37] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:37] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:37] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:37] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:37] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:37] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:45] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:45] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:45] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:45] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:45] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:45] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:45] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:45] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:49] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:49] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:49] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:49] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:49] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:49] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:49] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:49] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:54] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:54] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:54] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:54] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:54] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:54] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:54] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:08:54] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:09:01] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:09:01] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:09:01] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:09:01] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:09:01] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:09:01] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:09:01] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:09:01] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:09:09] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:09:09] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:09:09] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:09:09] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:09:09] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:09:09] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:09:09] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:09:09] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:09:13] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:09:13] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:09:13] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:09:13] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:09:13] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:09:13] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:09:13] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:09:13] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:09:18] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:09:18] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:09:18] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:09:18] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:09:18] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:09:18] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:09:18] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:09:18] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
cat(sprintf("Fitting time taken: %f minutes", 
            difftime(Sys.time(), start_time, units = "mins")))
#> Fitting time taken: 5.183153 minutes

# (html) table of accuracy metrics
model_results$errors %>%
  simChef::pretty_DT(digits = 2, sigfig = F, rownames = FALSE, 
           caption = "Validation Prediction Accuracies", 
           options = list(dom = 't'))

# (latex) table of accuracy metrics
model_results$errors %>%
  simChef::pretty_kable(digits = 2, sigfig = F, row.names = FALSE, 
           caption = "Validation Prediction Accuracies",
           format = "latex")

3.2 Stability check

Taking the prediction methods that pass the prediction check, perform stability analysis.

Specify and justify the appropriate data perturbation(s).
Re-fit the prediction methods on these perturbed data sets.
Evaluate prediction methods on validation data.
Assess stability across the data perturbations as well as across the various methods.
Filter out poor models where necessary and interpret stability results.

TODO: Add results for h20, tuning needs to be fixed for h20 + tidymodels

n_reps <- 2 # increase for better stability measures when not testing code

# p <- progressr::progressor(steps = n_reps)
# future::plan(multisession, workers = min(n_reps, parallel::detectCores() - 1))
bootstrap_model_results <- future.apply::future_replicate(
  n = n_reps,
  expr = {
    bootstrap <- sample(1:nrow(Xtrain), nrow(Xtrain), replace = TRUE)
    Xtrain_b <- Xtrain[bootstrap, ]
    ytrain_b <- ytrain[bootstrap]
    
    if (params$modeling_pkg == "caret") {
      model_results_b <- fitCaret(Xtrain = Xtrain_b, ytrain = ytrain_b, 
                                  Xtest = Xvalid, ytest = yvalid, 
                                  model_list = model_list, 
                                  tr_control = tr_control, 
                                  response_type = response_type)
    } else if (params$modeling_pkg == "tidymodels") {
      model_results_b <- fitTidyModels(Xtrain = Xtrain_b, ytrain = ytrain_b, 
                                       Xtest = Xvalid, ytest = yvalid,
                                       model_list = model_list, kfolds = kfolds)
    } else if (params$modeling_pkg == "h2o") {
      model_results_b <- fith2o(Xtrain = Xtrain_b, ytrain = ytrain_b,
                                Xtest = Xvalid, ytest = yvalid,
                                model_list = model_list)
    }
    return(model_results_b)
  },
  simplify = FALSE
)
#> [11:11:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:11:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:11:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:11:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:11:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:11:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:11:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:11:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:11:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:11:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:11:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:11:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:11:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:11:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:11:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:11:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:11:57] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:11:57] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:11:57] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:11:57] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:11:57] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:11:57] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:11:57] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:11:57] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:01] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:01] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:01] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:01] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:01] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:01] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:01] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:01] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:07] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:07] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:07] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:07] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:07] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:07] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:07] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:07] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:13] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:13] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:13] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:13] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:14] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:14] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:14] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:14] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:27] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:27] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:27] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:27] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:27] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:27] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:27] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:27] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:34] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:34] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:34] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:34] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:34] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:34] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:34] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:34] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:38] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:38] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:38] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:38] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:38] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:38] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:38] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:38] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:48] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:48] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:48] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:48] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:48] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:48] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:48] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:48] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:55] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:55] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:55] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:55] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:55] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:55] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:55] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:55] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:59] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:59] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:59] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:59] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:59] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:59] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:59] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:12:59] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:09] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:09] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:09] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:09] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:09] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:09] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:09] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:09] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:20] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:20] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:20] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:20] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:20] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:20] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:20] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:20] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:24] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:24] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:24] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:24] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:24] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:24] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:24] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:13:24] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:15:45] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:15:45] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:15:45] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:15:45] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:15:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:15:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:15:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:15:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:15:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:15:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:15:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:15:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:15:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:15:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:15:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:15:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:15:57] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:15:57] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:15:57] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:15:57] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:15:57] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:15:57] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:15:57] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:15:57] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:02] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:02] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:02] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:02] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:03] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:03] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:03] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:03] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:09] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:09] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:09] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:09] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:09] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:09] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:09] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:09] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:25] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:25] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:25] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:25] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:25] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:25] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:25] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:25] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:31] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:31] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:31] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:31] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:31] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:31] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:31] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:31] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:38] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:38] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:38] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:38] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:38] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:38] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:38] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:38] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:47] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:47] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:47] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:47] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:47] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:47] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:47] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:47] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:16:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:01] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:01] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:01] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:01] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:01] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:01] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:01] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:01] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:05] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:05] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:05] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:05] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:05] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:05] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:05] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:05] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:10] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:10] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:10] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:10] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:10] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:10] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:10] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:10] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:23] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:23] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:23] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:23] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:23] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:23] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:23] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:23] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:28] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:28] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:28] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:28] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:28] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:28] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:28] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:28] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:32] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:32] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:32] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:32] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:32] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:32] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:32] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:17:32] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.

bootstrap_model_errs <- purrr::map_dfr(bootstrap_model_results, "errors", 
                                       .id = "bootstrap_id")
bootstrap_model_preds <- purrr::map(bootstrap_model_results, "predictions",
                                    .id = "bootstrap_id")
bootstrap_model_imps <- purrr::map(bootstrap_model_results, "importance",
                                   .id = "bootstrap_id") %>%
  dplyr::bind_rows(.id = "bootstrap_id")

bootstrap_model_errs_summary <- bootstrap_model_errs %>%
  dplyr::group_by(model) %>%
  dplyr::summarise(dplyr::across(-bootstrap_id, list(mean = mean, sd = sd))) %>%
  setNames(stringr::str_replace(colnames(.), "_", " "))

# (html) table of accuracy metrics
bootstrap_model_errs_summary %>%
  simChef::pretty_DT(
    digits = 2, sigfig = F, rownames = FALSE, 
    caption = "Validation Prediction Accuracies Over Bootstrapped Training Fits", 
    bold_function = ". == max(., na.rm = TRUE)", bold_margin = 2,
    bold_scheme = c(F, T, F, T, F),
    options = list(dom = 't')
  )

# (latex) table of accuracy metrics
bootstrap_model_errs_summary %>%
  simChef::pretty_kable(
    digits = 2, sigfig = F, row.names = FALSE, 
    caption = "Validation Prediction Accuracies Over Bootstrapped Training Fits", 
    bold_function = ". == max(., na.rm = TRUE)", bold_margin = 2,
    bold_scheme = c(F, T, F, T, F),
    format = "latex"
  )

# boxplots
bootstrap_model_errs %>%
  tidyr::pivot_longer(cols = -c(model, bootstrap_id), 
                      names_to = "Metric", values_to = "Value") %>%
  plotBoxplot(x_str = "Value", y_str = "model", horizontal = FALSE) +
  ggplot2::facet_wrap(~ Metric) +
  ggplot2::labs(x = "Method", 
                title = "Validation Prediction Accuracies Over Bootstrapped Training Fits")

3.3 Interpretability

For the models that pass the prediction and stability checks, extract the important features in the predictive models that are stable across both data and model perturbations. Determining the importance of a feature can be method dependent.

Full Model (without stability)

Table

simChef::pretty_DT(model_results$importance, digits = 2, sigfig = F, 
                   caption = "Variable Importances")

simChef::pretty_kable(model_results$importance, digits = 2, sigfig = F, 
                      caption = "Variable Importances", format = "latex")

Plots

plotFeatureImportance(model_results$importance,
                      use_rankings = FALSE,
                      use_facets = TRUE,
                      interactive = FALSE)

plotFeatureImportancePair(model_results$importance,
                          use_rankings = TRUE,
                          interactive = FALSE)

Bootstrapped Model (with stability)

Table

bootstrap_model_imps_summary <- bootstrap_model_imps %>%
  dplyr::group_by(model, variable) %>%
  dplyr::summarise(`Mean Importance` = mean(importance),
                   `Median Importance` = median(importance),
                   `SD Importance` = sd(importance),
                   `Min Importance` = min(importance),
                   `Max Importance` = max(importance))
simChef::pretty_DT(bootstrap_model_imps_summary, 
                   digits = 2, sigfig = F, 
                   caption = "Summary of variable importances across bootstrapped models")

bootstrap_model_imps_summary <- bootstrap_model_imps %>%
  dplyr::group_by(model, variable) %>%
  dplyr::summarise(`Mean Importance` = mean(importance),
                   `Median Importance` = median(importance),
                   `SD Importance` = sd(importance),
                   `Min Importance` = min(importance),
                   `Max Importance` = max(importance))
simChef::pretty_kable(bootstrap_model_imps_summary, 
                      digits = 2, sigfig = F, 
                      caption = "Summary of variable importances across bootstrapped models",
                      format = "latex")

Plots

plotFeatureImportanceStability(bootstrap_model_imps,
                               use_rankings = FALSE,
                               use_facets = FALSE,
                               interactive = FALSE)

4 Main Results

Interpret and summarize the prediction and stability results.

Evaluate pipeline on test data.

Xtrain_final <- dplyr::bind_rows(Xtrain, Xvalid)
ytrain_final <- c(ytrain, yvalid)
if (params$modeling_pkg == "caret") {
  final_model_results <- fitCaret(Xtrain = Xtrain_final, ytrain = ytrain_final, 
                                  Xtest = Xtest, ytest = ytest, 
                                  model_list = model_list, 
                                  tr_control = tr_control, 
                                  response_type = response_type)
} else if (params$modeling_pkg == "tidymodels") {
  final_model_results <- fitTidyModels(Xtrain = Xtrain_final, 
                                       ytrain = ytrain_final, 
                                       Xtest = Xtest, ytest = ytest,
                                       model_list = model_list, kfolds = kfolds)
} else if (params$modeling_pkg == "h2o") {
  final_model_results <- fith2o(Xtrain = Xtrain_final, ytrain = ytrain_final,
                                Xtest = Xtest, ytest = ytest,
                                model_list = model_list)
}
#> [11:22:24] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:24] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:24] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:24] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:24] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:24] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:24] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:24] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:35] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:35] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:35] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:35] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:35] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:35] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:35] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:35] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:41] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:41] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:41] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:41] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:48] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:48] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:48] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:48] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:48] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:48] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:48] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:48] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:57] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:57] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:57] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:57] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:57] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:57] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:57] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:22:57] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:09] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:09] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:09] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:09] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:09] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:09] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:09] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:09] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:15] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:15] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:15] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:15] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:15] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:15] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:15] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:15] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:22] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:22] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:22] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:22] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:22] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:22] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:22] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:22] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:31] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:31] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:31] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:31] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:31] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:31] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:31] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:31] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:49] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:49] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:49] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:49] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:49] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:49] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:49] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:49] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:55] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:55] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:55] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:55] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:55] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:55] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:55] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:23:55] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:15] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:15] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:15] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:15] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:15] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:15] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:15] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:15] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:27] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:27] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:27] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:27] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:27] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:27] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:27] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:27] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:36] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:36] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:36] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:36] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:36] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:36] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:36] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:36] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:47] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:47] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:47] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:47] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:47] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:47] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:47] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:47] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:59] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:59] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:59] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:59] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:59] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:59] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:59] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [11:24:59] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.

Summarize test set prediction and/or interpretability results.

# (html) table of accuracy metrics
final_model_results$errors %>%
  simChef::pretty_DT(digits = 2, sigfig = F, rownames = FALSE, 
           caption = "Test Prediction Accuracies", 
           options = list(dom = 't'))

# (latex) table of accuracy metrics
final_model_results$errors %>%
    simChef::pretty_kable(digits = 2, sigfig = F, row.names = FALSE, 
             caption = "Test Prediction Accuracies", format = "latex")

5 Post hoc analysis

Move beyond the global prediction accuracy metrics and dive deeper into individual-level predictions for the validation and/or test set, i.e., provide a more “local” analysis.

Examine any points that had poor predictions.
Examine differences between prediction methods.

plotPairs(cbind(`True Repsonses` = ytest, final_model_results$predictions), 
          columns = 1:(ncol(final_model_results$predictions) + 1), 
          title = "Comparison of model predictions")

6 Conclusions

Reiterate main findings, note any caveats, and clearly translate findings/analysis back to the domain problem context.

LS0tCnRpdGxlOiAiUENTIERvY3VtZW50YXRpb24iCmF1dGhvcjogIiIKZGF0ZTogImByIGZvcm1hdChTeXMudGltZSgpLCAnJUIgJWQsICVZJylgIgpoZWFkZXItaW5jbHVkZXM6CiAgICAtIFx1c2VwYWNrYWdle2Zsb2F0fQogICAgLSBcdXNlcGFja2FnZXthbXNtYXRofQogICAgLSBcdXNlcGFja2FnZXtnZW5zeW1ifQogICAgLSBcdXNlcGFja2FnZXt0Y29sb3Jib3h9Cm91dHB1dDoKICBybWRmb3JtYXRzOjptYXRlcmlhbDoKICAgIGZpZ19jYXB0aW9uOiB0cnVlCiAgICBjb2RlX2ZvbGRpbmc6IGhpZGUKICAgIG51bWJlcl9zZWN0aW9uczogdHJ1ZQogICAgdXNlX2Jvb2tkb3duOiB0cnVlCiAgICBmaWdfd2lkdGg6IDEwCiAgICBmaWdfaGVpZ2h0OiA4CiAgICBsaWdodGJveDogdHJ1ZQogICAgY29kZV9kb3dubG9hZDogdHJ1ZQogICAgaW5jbHVkZXM6CiAgICAgIGJlZm9yZV9ib2R5OiBodG1sL3NldHVwLmh0bWwKICBwZGZfZG9jdW1lbnQ6CiAgICBudW1iZXJfc2VjdGlvbnM6IHRydWUKcGFyYW1zOgogIFhfZmlsZXBhdGg6CiAgICBsYWJlbDogIlggRGF0YSIKICAgICMgdmFsdWU6IE5VTEwKICAgIHZhbHVlOiBkYXRhL3RjZ2FfYnJjYV9hcnJheV9kYXRhLnJkcwogICAgaW5wdXQ6IGZpbGUgCiAgeV9maWxlcGF0aDoKICAgIGxhYmVsOiAieSBkYXRhIgogICAgIyB2YWx1ZTogTlVMTAogICAgdmFsdWU6IGRhdGEvdGNnYV9icmNhX3N1YnR5cGVzLnJkcwogICAgaW5wdXQ6IGZpbGUKICB0cmFpbl9wcm9wOgogICAgbGFiZWw6ICJUcmFpbmluZyBkYXRhIHByb3BvcnRpb24iCiAgICB2YWx1ZTogMC42CiAgICBpbnB1dDogbnVtZXJpYwogIHZhbGlkX3Byb3A6CiAgICBsYWJlbDogIlZhbGlkYXRpb24gZGF0YSBwcm9wb3J0aW9uIgogICAgdmFsdWU6IDAuMgogICAgaW5wdXQ6IG51bWVyaWMKICB0ZXN0X3Byb3A6CiAgICBsYWJlbDogIlRlc3QgZGF0YSBwcm9wb3J0aW9uIgogICAgdmFsdWU6IDAuMgogICAgaW5wdXQ6IG51bWVyaWMKICBtb2RlbGluZ19wa2c6CiAgICBsYWJlbDogIk1vZGVsaW5nIFBhY2thZ2UiCiAgICB2YWx1ZTogImNhcmV0IgogICAgaW5wdXQ6IHNlbGVjdAogICAgY2hvaWNlczogWyJjYXJldCIsICJoMm8iLCAidGlkeW1vZGVscyJdCiAgc2VlZDoKICAgIGxhYmVsOiAiUmFuZG9tIFNlZWQiCiAgICB2YWx1ZTogMTIzNDUKICAgIGlucHV0OiBudW1lcmljCmNzczogW2Nzcy9jdXN0b21fbWF0ZXJpYWxfcm1kX3RoZW1lLmNzcywgY3NzL2N1c3RvbV9jaGVja2JveC5jc3NdCiMgcnVudGltZTogc2hpbnkKLS0tCgpgYGB7ciBzZXR1cCwgZWNobyA9IEZBTFNFLCBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQpzb3VyY2UoInNjcmlwdHMvcm1kLXNldHVwLlIiLCBsb2NhbCA9IGtuaXRyOjprbml0X2dsb2JhbCgpKQoKIyBsb2FkIGluIHBhY2thZ2VzCmxpYnJhcnkodGlkeXZlcnNlKQpsaWJyYXJ5KHNraW1yKQpsaWJyYXJ5KFIudXRpbHMpCmxpYnJhcnkoZGF0YXNldHMpCmxpYnJhcnkoZ2dwbG90MikKbGlicmFyeShkcGx5cikKbGlicmFyeSh0aWR5cikKbGlicmFyeShyZXNoYXBlMikKZm9yIChmIGluIGxpc3QuZmlsZXMoIlIiLCBwYXR0ZXJuID0gIi5SIiwgZnVsbC5uYW1lcyA9IFRSVUUpKSB7CiAgc291cmNlKGYsIGNoZGlyID0gVCkKfQoKIyBzZXQgc2VlZApzZXQuc2VlZChwYXJhbXMkc2VlZCkKCiMgdmFsaWRhdGUgaW5wdXRzCnZhbGlkYXRlRGF0YVNwbGl0KHBhcmFtcyR0cmFpbl9wcm9wLCBwYXJhbXMkdmFsaWRfcHJvcCwgcGFyYW1zJHRlc3RfcHJvcCkKaWYgKHBhcmFtcyRtb2RlbGluZ19wa2cgIT0gImNhcmV0IikgewogIHN0b3AoIk9ubHkgY2FyZXQgbW9kZWxpbmcgYmFja2VuZCBoYXMgYmVlbiBpbXBsZW1lbnRlZC4iKQp9CgojIGxvYWQgZGF0YSBpbgpYIDwtIHVwbG9hZEZpbGUocGFyYW1zJFhfZmlsZXBhdGgpCnkgPC0gdXBsb2FkRmlsZShwYXJhbXMkeV9maWxlcGF0aCkKI1RPRE86IHZhbGlkYXRlIGRhdGEgYmV0dGVyCnZhbGlkYXRlRGF0YShYLCB5KQoKIyBpbml0aWFsaXplIGNvdW50ZXIgZm9yIHN1YmNodW5raWZ5CmNodW5rX2lkeCA8LSAxCmBgYAoKIyBEb21haW4gcHJvYmxlbSBmb3JtdWxhdGlvbgoKV2hhdCBpcyB0aGUgcmVhbC13b3JsZCBxdWVzdGlvbj8gVGhpcyBjb3VsZCBiZSBoeXBvdGhlc2lzLWRyaXZlbiBvciBkaXNjb3ZlcnktYmFzZWQuCgpgYGB7YXNpcywgaGVscCA9IFRSVUV9ClNvbWUgYWR2aWNlCmBgYAoKYGBge2FzaXMsIGludGVyYWN0aXZlX3RleHQgPSBUUlVFfQoKYGBgCgpXaHkgaXMgdGhpcyBxdWVzdGlvbiBpbnRlcmVzdGluZyBhbmQgaW1wb3J0YW50PyBXaGF0IGFyZSB0aGUgaW1wbGljYXRpb25zIG9mIGJldHRlciB1bmRlcnN0YW5kaW5nIHRoaXMgZGF0YT8KCmBgYHthc2lzLCBpbnRlcmFjdGl2ZV90ZXh0ID0gVFJVRX0KCmBgYAoKQnJpZWZseSBkZXNjcmliZSBhbnkgYmFja2dyb3VuZCBpbmZvcm1hdGlvbiBuZWNlc3NhcnkgdG8gdW5kZXJzdGFuZCB0aGlzIHByb2JsZW0uCgpgYGB7YXNpcywgaGVscCA9IFRSVUV9ClNvbWUgYWR2aWNlCmBgYAoKYGBge2FzaXMsIGludGVyYWN0aXZlX3RleHQgPSBUUlVFfQoKYGBgCgpCcmllZmx5IGRlc2NyaWJlIGhvdyB0aGlzIHF1ZXN0aW9uIGNhbiBiZSBhbnN3ZXJlZCBpbiB0aGUgY29udGV4dCBvZiBhIG1vZGVsIG9yIGFuYWx5c2lzLgoKYGBge2FzaXMsIGludGVyYWN0aXZlX3RleHQgPSBUUlVFfQoKYGBgCgpPdXRsaW5lIHRoZSByZXN0IG9mIHRoZSByZXBvcnQvYW5hbHlzaXMuCgpgYGB7YXNpcywgaW50ZXJhY3RpdmVfdGV4dCA9IFRSVUV9CgpgYGAKCiMgRGF0YQoKV2hhdCBpcyB0aGUgZGF0YSB1bmRlciBpbnZlc3RpZ2F0aW9uPyBQcm92aWRlIGEgYnJpZWYgb3ZlcnZpZXcvZGVzY3JpcHRpb24gb2YgdGhlIGRhdGEuCgpgYGB7YXNpcywgaW50ZXJhY3RpdmVfdGV4dCA9IFRSVUV9CgpgYGAKCkRlc2NyaWJlIGhvdyB5b3VyIGRhdGEgY29ubmVjdHMgdG8gdGhlIGRvbWFpbiBwcm9ibGVtLgoKYGBge2FzaXMsIGludGVyYWN0aXZlX3RleHQgPSBUUlVFfQoKYGBgCgojIyBEYXRhIENvbGxlY3Rpb24KCkhvdyB3YXMgdGhlIGRhdGEgY29sbGVjdGVkIG9yIGdlbmVyYXRlZCAoaW5jbHVkaW5nIGRldGFpbHMgb24gdGhlIGV4cGVyaW1lbnRhbCBkZXNpZ24pPyBCZSBhcyB0cmFuc3BhcmVudCBhcyBwb3NzaWJsZSBzbyB0aGF0IGNvbmNsdXNpb25zIG1hZGUgZnJvbSB0aGlzIGRhdGEgYXJlIG5vdCBtaXNpbnRlcnByZXRlZCBkb3duIHRoZSByb2FkLgoKYGBge2FzaXMsIGludGVyYWN0aXZlX3RleHQgPSBUUlVFfQoKYGBgCgpEZXNjcmliZSBhbnkgbGltaXRhdGlvbnMgd2hlbiB1c2luZyB0aGUgZGF0YSB0byBhbnN3ZXIgdGhlIGRvbWFpbiBwcm9ibGVtIG9mIGludGVyZXN0LgoKYGBge2FzaXMsIGludGVyYWN0aXZlX3RleHQgPSBUUlVFfQoKYGBgCgpXaGVyZSBpcyB0aGUgZGF0YSBzdG9yZWQsIGFuZCBob3cgY2FuIGl0IGJlIGFjY2Vzc2VkIGJ5IG90aGVycyAoaWYgYXBwbGljYWJsZSk/CgpgYGB7YXNpcywgaW50ZXJhY3RpdmVfdGV4dCA9IFRSVUV9CgpgYGAKCiMjIERhdGEgU3BsaXR0aW5nCgpUT0RPOiBhZGQgYWR2aWNlIGZvciBwb3NzaWJsZSBkYXRhIHNwbGl0cywgQUsgZ2V0dGluZyBuaWNlIGZpZ3VyZSB0b2dldGhlcgoKRGVjaWRlIG9uIHRoZSBwcm9wb3J0aW9uIG9mIGRhdGEgaW4gZWFjaCBzcGxpdC4KCkRlY2lkZSBvbiB0aGUgImhvdyIgdG8gc3BsaXQgdGhlIGRhdGEgKGUuZy4sIHJhbmRvbSBzYW1wbGluZywgc3RyYXRpZmllZCBzYW1wbGluZywgZXRjLiksIGFuZCBleHBsYWluIHdoeSB0aGlzIGlzIGEgcmVhc29uYWJsZSB3YXkgdG8gc3BsaXQgdGhlIGRhdGEuCgpgYGB7YXNpcywgaW50ZXJhY3RpdmVfdGV4dCA9IFRSVUV9CgpgYGAKClNwbGl0IHRoZSBkYXRhIGludG8gYSB0cmFpbmluZywgdmFsaWRhdGlvbiwgYW5kIHRlc3Qgc2V0LgoKYGBge3Igc3BsaXQtZGF0YX0KZGF0YV9zcGxpdCA8LSBiYXNpY0RhdGFTcGxpdChYID0gWCwgeSA9IHksIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgIHRyYWluX3Byb3AgPSBwYXJhbXMkdHJhaW5fcHJvcCwgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdmFsaWRfcHJvcCA9IHBhcmFtcyR2YWxpZF9wcm9wLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICB0ZXN0X3Byb3AgPSBwYXJhbXMkdGVzdF9wcm9wLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgIHN0cmF0aWZpZWRfYnkgPSB5KQpYdHJhaW4gPC0gZGF0YV9zcGxpdCRYJHRyYWluClh2YWxpZCA8LSBkYXRhX3NwbGl0JFgkdmFsaWRhdGUKWHRlc3QgPC0gZGF0YV9zcGxpdCRYJHRlc3QKeXRyYWluIDwtIGRhdGFfc3BsaXQkeSR0cmFpbgp5dmFsaWQgPC0gZGF0YV9zcGxpdCR5JHZhbGlkYXRlCnl0ZXN0IDwtIGRhdGFfc3BsaXQkeSR0ZXN0CmBgYAoKUHJvdmlkZSBzdW1tYXJ5IHN0YXRpc3RpY3MgYW5kL29yIGZpZ3VyZXMgb2YgdGhlIHRocmVlIGRhdGEgc2V0cyB0byBpbGx1c3RyYXRlIGhvdyBzaW1pbGFyIChvciBkaWZmZXJlbnQpIHRoZXkgYXJlLgoKYGBge2FzaXMsIGludGVyYWN0aXZlX3RleHQgPSBUUlVFfQoKYGBgCgojIyMgRGF0YSBTcGxpdHRpbmcgT3ZlcnZpZXcgey50YWJzZXQgLmN1c3RvbS10YWJzfQoKIyMjIyBYIERhdGEgU3BsaXQgey51bm51bWJlcmVkfQoKYGBge3IgWC1kYXRhLXNwbGl0LCBhZGRfbmV3X2xpbmUgPSBUUlVFfQpwbG90X1hfZGF0YXNwbGl0KFh0cmFpbiwgWHZhbGlkLCBYdGVzdCkKYGBgCgojIyMjIFkgRGF0YSBTcGxpdCB7LnVubnVtYmVyZWR9CgpgYGB7ciB5LWRhdGEtc3BsaXQsIGFkZF9uZXdfbGluZSA9IFRSVUV9CnBsb3RfeV9kYXRhc3BsaXQoeXRyYWluLCB5dmFsaWQsIHl0ZXN0KQpgYGAKCiMjIERhdGEgQ2xlYW5pbmcgYW5kIFByZXByb2Nlc3NpbmcKCldoYXQgc3RlcHMgd2VyZSB0YWtlbiB0byBjbGVhbiB0aGUgZGF0YT8gTW9yZSBpbXBvcnRhbnRseSwgd2h5IHdhcyB0aGUgZGF0YSBjbGVhbmVkIGluIHRoaXMgd2F5PwoKRGlzY3VzcyBhbGwgaW5jb25zaXN0ZW5jaWVzLCBwcm9ibGVtcywgb2RkaXRpZXMgaW4gdGhlIGRhdGEgKGUuZy4sIG1pc3NpbmcgZGF0YSwgZXJyb3JzIGluIGRhdGEsIG91dGxpZXJzLCBldGMuKS4KClJlY29yZCB5b3VyIHByZXByb2Nlc3Npbmcgc3RlcHMgaW4gYSB3YXkgc3VjaCB0aGF0IGlmIHNvbWVvbmUgZWxzZSB3ZXJlIHRvIHJlcHJvZHVjZSB5b3VyIGFuYWx5c2lzLCB0aGV5IGNvdWxkIGVhc2lseSByZXBsaWNhdGUgYW5kIHVuZGVyc3RhbmQgeW91ciBzdGVwcy4KCkl0IGNhbiBiZSBoZWxwZnVsIHRvIGluY2x1ZGUgcmVsZXZhbnQgcGxvdHMgdGhhdCBleHBsYWluL2p1c3RpZnkgdGhlIGNob2ljZXMgdGhhdCB3ZXJlIG1hZGUgd2hlbiBjbGVhbmluZyB0aGUgZGF0YS4KCklmIG1vcmUgdGhhbiBvbmUgcHJlcHJvY2Vzc2luZyBwaXBlbGluZSBpcyByZWFzb25hYmxlLCBleGFtaW5lIHRoZSBpbXBhY3RzIG9mIHRoZXNlIGFsdGVybmF0aXZlIHByZXByb2Nlc3NpbmcgcGlwZWxpbmVzIG9uIHRoZSBmaW5hbCBkYXRhIHJlc3VsdHMuCgpBZ2FpbiwgYmUgYXMgdHJhbnNwYXJlbnQgYXMgcG9zc2libGUuIFRoaXMgYWxsb3dzIG90aGVycyB0byBtYWtlIHRoZWlyIG93biBlZHVjYXRlZCBkZWNpc2lvbnMgb24gaG93IGJlc3QgdG8gcHJlcHJvY2VzcyB0aGUgZGF0YS4KCmBgYHthc2lzLCBpbnRlcmFjdGl2ZV90ZXh0ID0gVFJVRX0KCmBgYAoKYGBge3IgcHJlcHJvY2Vzcy1kYXRhfQpYdHJhaW4gPC0gbG9nKFh0cmFpbiArIDEpICU+JQogIHJlbW92ZUNvbnN0YW50Q29scyh2ZXJib3NlID0gMSkgJT4lCiAgcmVtb3ZlRHVwbGljYXRlQ29scyh2ZXJib3NlID0gMSkgJT4lCiAgZmlsdGVyQnlWYXIobWF4LnAgPSAxMDAwKQpYdmFsaWQgPC0gbG9nKFh2YWxpZCArIDEpWywgY29sbmFtZXMoWHRyYWluKV0KWHRlc3QgPC0gbG9nKFh0ZXN0ICsgMSlbLCBjb2xuYW1lcyhYdHJhaW4pXQpgYGAKCgojIyBEYXRhIEV4cGxvcmF0aW9uCgpUT0RPOiBBZGQgZHJhZyBhbmQgZHJvcCBmZWF0dXJlIGluIHNoaW55IHZlcnNpb24gZm9yIG90aGVyIGltYWdlcwoKVGhlIG1haW4gZ29hbCBvZiB0aGlzIHNlY3Rpb24gaXMgdG8gZ2l2ZSB0aGUgcmVhZGVyIGEgZmVlbCBmb3Igd2hhdCB0aGUgZGF0YSAibG9va3MgbGlrZSIgYXQgYSBiYXNpYyBsZXZlbC4KClByb3ZpZGUgcGxvdHMgdGhhdCBzdW1tYXJpemUgdGhlIGRhdGEgYW5kIHBlcmhhcHMgZXZlbiBwbG90cyB0aGF0IGNvbnZleSBzb21lIHNtYWxsZXIgZmluZGluZ3Mgd2hpY2ggdWx0aW1hdGVseSBtb3RpdmF0ZSB0aGUgbWFpbiBmaW5kaW5ncy4KClByb3ZpZGUgYWRkaXRpb25hbCBwbG90cyByZXByZXNlbnRpbmcgcmVtYWluaW5nIG9kZGl0aWVzIGFmdGVyIHByZS1wcm9jZXNzaW5nIGlmIGFwcGxpY2FibGUuCgpBZGQgc3VtbWFyeSBzdGF0aXN0aWNzIGluIGFjY29tcGFueWluZyB0YWJsZXMgKG9yIGluIGZpZ3VyZXMpIGZvciBxdWljayBjb21wYXJpc29ucy4KCmBgYHthc2lzLCBpbnRlcmFjdGl2ZV90ZXh0ID0gVFJVRX0KCmBgYAoKIyMjIFRyYWluaW5nIERhdGEgT3ZlcnZpZXcgey50YWJzZXQgLmN1c3RvbS10YWJzfQoKYGBge3IgZWNobyA9IEZBTFNFfQpkYXRhT3ZlcnZpZXcoWHRyYWluID0gWHRyYWluLCB5dHJhaW4gPSB5dHJhaW4pICU+JQogIGNhdCgpCmBgYAoKIyMjIyBTdW1tYXJ5IFRhYmxlcyB7LnVubnVtYmVyZWR9CgpgYGB7ciByZXN1bHRzPSJhc2lzIiwgYWRkX25ld19saW5lID0gVFJVRX0KIyBzdW1tYXJ5IG9mIHR5cGVzIG9mIGZlYXR1cmVzIGluIHRyYWluaW5nIFgKZGF0YV90eXBlcyhYdHJhaW4gPSBYdHJhaW4sIHl0cmFpbiA9IHl0cmFpbikKYGBgCgpgYGB7ciByZXN1bHRzPSJhc2lzIiwgYWRkX25ld19saW5lID0gVFJVRX0KIyBzdW1tYXJ5IG9mIGZlYXR1cmVzIGluIHRyYWluaW5nIFgsIHNvcnRlZCBieSB0eXBlCnRhYl9scyA8LSBkYXRhX3N1bW1hcnkoWHRyYWluID0gWHRyYWluLCB5dHJhaW4gPSB5dHJhaW4sIGRpZ2l0cyA9IDIsIHNpZ2ZpZyA9IEYpCmZvciAoZHRfbmFtZSBpbiBuYW1lcyh0YWJfbHMpKSB7CiAgc2ltQ2hlZjo6OnN1YmNodW5raWZ5KHRhYl9sc1tbZHRfbmFtZV1dLCBpID0gY2h1bmtfaWR4LCAKICAgICAgICAgICAgICAgICAgICAgICAgb3RoZXJfYXJncyA9ICJyZXN1bHRzPSdhc2lzJyIpCiAgY2h1bmtfaWR4IDwtIGNodW5rX2lkeCArIDEKfQpgYGAKCiMjIyMgWCBkaXN0cmlidXRpb24gey51bm51bWJlcmVkfQoKYGBge3IsIGFkZF9uZXdfbGluZSA9IFRSVUV9CiMgcGxvdCBYIGRpc3RyaWJ1dGlvbgpwbG90X1hfZGlzdHJpYnV0aW9uKFh0cmFpbiwgImRlbnNpdHkiKQpgYGAKCiMjIyMgWSBkaXN0cmlidXRpb24gey51bm51bWJlcmVkfQoKYGBge3IsIGFkZF9uZXdfbGluZSA9IFRSVUV9CiMgcGxvdCB5IGRpc3RyaWJ1dGlvbgpwbG90X3lfZGlzdHJpYnV0aW9uKHl0cmFpbiwgImJhciIpCmBgYAoKIyMjIyBGZWF0dXJlIENvcnJlbGF0aW9uIHsudW5udW1iZXJlZH0KCmBgYHtyLCBhZGRfbmV3X2xpbmUgPSBUUlVFfQojIGNvcnJlbGF0aW9uIGhlYXRtYXAKa2VlcF9mZWF0dXJlcyA8LSBzb3J0KAogIHNhbXBsZSgxOm5jb2woWHRyYWluKSwgbWluKG5jb2woWHRyYWluKSwgMTAwMCksIHJlcGxhY2UgPSBGQUxTRSkKKQpwbG90Q29ySGVhdG1hcChYID0gWHRyYWluWywga2VlcF9mZWF0dXJlc10sIGNvcl90eXBlID0gInBlYXJzb24iLCAKICAgICAgICAgICAgICAgICAgICAgIGNsdXN0ID0gVFJVRSwgdGV4dF9zaXplID0gMCkgKwogIGdncGxvdDI6OmxhYnMoeCA9ICJGZWF0dXJlcyIsIHkgPSAiRmVhdHVyZXMiLCBmaWxsID0gIkNvci4iKSArCiAgc2ltQ2hlZjo6cHJldHR5X2dncGxvdF90aGVtZSh4X3RleHRfYW5nbGUgPSBUUlVFKQpgYGAKCiMjIyMgRmVhdHVyZSBQYWlyIFBsb3RzIHsudW5udW1iZXJlZH0KCmBgYHtyLCBhZGRfbmV3X2xpbmUgPSBUUlVFfQojIHBhaXIgcGxvdHMKa2VlcF9mZWF0dXJlcyA8LSBzb3J0KAogIHNhbXBsZSgxOm5jb2woWHRyYWluKSwgbWluKG5jb2woWHRyYWluKSwgNiksIHJlcGxhY2UgPSBGQUxTRSkKKQpwbG90UGFpcnMoZGF0YSA9IFh0cmFpbiwgY29sdW1ucyA9IGtlZXBfZmVhdHVyZXMsIAogICAgICAgICAgY29sb3IgPSB5dHJhaW4sIGNvbG9yX2xhYmVsID0gInkiKQpgYGAKCiMjIyMgTWFyZ2luYWwgQXNzb2NpYXRpb25zIHsudW5udW1iZXJlZH0KCmBgYHtyLCBhZGRfbmV3X2xpbmUgPSBUUlVFfQpjYXJldDo6ZmVhdHVyZVBsb3QoeCA9IFh0cmFpblssIGtlZXBfZmVhdHVyZXNdLAogICAgICAgICAgICAgICAgICAgeSA9IHl0cmFpbiwKICAgICAgICAgICAgICAgICAgIHBsb3QgPSBpZiAoaXMuZmFjdG9yKHl0cmFpbikpICJib3giIGVsc2UgInNjYXR0ZXIiLAogICAgICAgICAgICAgICAgICAgIyBzdHJpcCA9IHN0cmlwLmN1c3RvbShwYXIuc3RyaXAudGV4dCA9IGxpc3QoY2V4ID0gLjcpKSwKICAgICAgICAgICAgICAgICAgIHNjYWxlcyA9IGxpc3QoeCA9IGxpc3QocmVsYXRpb24gPSAiZnJlZSIpLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgeSA9IGxpc3QocmVsYXRpb24gPSAiZnJlZSIpKSkKYGBgCgojIyMjIFBDQSB7LnVubnVtYmVyZWR9CgpgYGB7ciwgYWRkX25ld19saW5lID0gVFJVRX0KIyBkaW1lbnNpb24gcmVkdWN0aW9uIHBsb3RzCnBsb3RQQ0EoWCA9IFh0cmFpbiwgbnBjcyA9IDMsIGNvbG9yID0geXRyYWluLCBjb2xvcl9sYWJlbCA9ICJ5IiwKICAgICAgICBjZW50ZXIgPSBULCBzY2FsZSA9IEZBTFNFKSRwbG90CmBgYAoKIyBQcmVkaWN0aW9uIE1vZGVsaW5nCgpUT0RPOiBhZGQgYWR2aWNlIG9uIHdoaWNoIG1vZGVscyB0byBzZWxlY3QgYW5kIHdoeQoKRGlzY3VzcyB0aGUgcHJlZGljdGlvbiBtZXRob2RzIHVuZGVyIGNvbnNpZGVyYXRpb24sIGFuZCBleHBsYWluIHdoeSB0aGVzZSBtZXRob2RzIHdlcmUgY2hvc2VuLgoKYGBge2FzaXMsIGludGVyYWN0aXZlX3RleHQgPSBUUlVFfQoKYGBgCgpEaXNjdXNzIHRoZSBhY2N1cmFjeSBtZXRyaWNzIHVuZGVyIGNvbnNpZGVyYXRpb24sIGFuZCBleHBsYWluIHdoeSB0aGVzZSBtZXRyaWNzIHdlcmUgY2hvc2VuLgoKYGBge2FzaXMsIGludGVyYWN0aXZlX3RleHQgPSBUUlVFfQoKYGBgCgpOb3RlOiB0aGVyZSBzaG91bGQgYmUgbXVsdGlwbGUgbWV0aG9kcyBhbmQgbWV0cmljcyB1bmRlciBjb25zaWRlcmF0aW9uIHRvIHBhaW50IGEgbW9yZSBob2xpc3RpYyBwaWN0dXJlIG9mIHRoZSBkYXRhLiBBdCBsZWFzdCBvbmUgbWV0aG9kIHNob3VsZCBiZSBhIGJhc2VsaW5lLCBjb21tb24gYXBwcm9hY2ggdGhhdCBtYXkgbm90IGJlIG9wdGltYWwgZm9yIHRoZSBwcm9ibGVtIHNldHRpbmcsIGJ1dCBzZXJ2ZXMgYXMgYSBoZWxwZnVsIGNvbXBhcmlzb24uCgojIyBQcmVkaWN0aW9uIGNoZWNrCgpDYXJyeSBvdXQgdGhlIHByZWRpY3Rpb24gcGlwZWxpbmUsIG91dGxpbmVkIGFib3ZlLgoKLSAgIEZpdCBwcmVkaWN0aW9uIG1ldGhvZHMgb24gdHJhaW5pbmcgZGF0YS4KLSAgIEV2YWx1YXRlIHByZWRpY3Rpb24gbWV0aG9kcyBvbiB2YWxpZGF0aW9uIGRhdGEuCi0gICBDb21wYXJlIHJlc3VsdHMsIGFuZCBmaWx0ZXIgb3V0IHBvb3IgbW9kZWxzLgoKYGBge2FzaXMsIGludGVyYWN0aXZlX3RleHQgPSBUUlVFfQoKYGBgCgpgYGB7ciBjYXJldC1maXRzLCBldmFsID0gcGFyYW1zJG1vZGVsaW5nX3BrZyA9PSAiY2FyZXQiLCBlY2hvID0gcGFyYW1zJG1vZGVsaW5nX3BrZyA9PSAiY2FyZXQifQojIyBQaWNrIG1vZGVscyBhbmQgZml0dGluZyBiYWNrZW5kIGZvciBjYXJldAp0cl9jb250cm9sIDwtIGNhcmV0Ojp0cmFpbkNvbnRyb2woCiAgbWV0aG9kID0gImN2IiwKICBudW1iZXIgPSA1LAogIGNsYXNzUHJvYnMgPSBpZiAoaXMuZmFjdG9yKHl0cmFpbikpIFRSVUUgZWxzZSBGQUxTRSwKICBzdW1tYXJ5RnVuY3Rpb24gPSBjYXJldDo6ZGVmYXVsdFN1bW1hcnksCiAgYWxsb3dQYXJhbGxlbCA9IEZBTFNFLAogIHZlcmJvc2VJdGVyID0gRkFMU0UKKQoKcmVzcG9uc2VfdHlwZSA8LSAicmF3Igptb2RlbF9saXN0IDwtIGxpc3QoCiAgcmFuZ2VyID0gbGlzdCh0dW5lR3JpZCA9IGV4cGFuZC5ncmlkKG10cnkgPSBzZXEoc3FydChuY29sKFh0cmFpbikpLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBuY29sKFh0cmFpbikgLyAzLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGxlbmd0aC5vdXQgPSAzKSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgc3BsaXRydWxlID0gImdpbmkiLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBtaW4ubm9kZS5zaXplID0gMSksCiAgICAgICAgICAgICAgICBpbXBvcnRhbmNlID0gImltcHVyaXR5IiwKICAgICAgICAgICAgICAgIG51bS50aHJlYWRzID0gMSksCiAgeGdiVHJlZSA9IGxpc3QodHVuZUdyaWQgPSBleHBhbmQuZ3JpZChucm91bmRzID0gYygxMCwgMjUsIDUwLCAxMDAsIDE1MCksCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBtYXhfZGVwdGggPSBjKDMsIDYpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgY29sc2FtcGxlX2J5dHJlZSA9IDAuMzMsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBldGEgPSBjKDAuMSwgMC4zKSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGdhbW1hID0gMCwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIG1pbl9jaGlsZF93ZWlnaHQgPSAxLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgc3Vic2FtcGxlID0gMC42KSwKICAgICAgICAgICAgICAgICBudGhyZWFkID0gMSkKKQoKc3RhcnRfdGltZSA8LSBTeXMudGltZSgpCm1vZGVsX3Jlc3VsdHMgPC0gZml0Q2FyZXQoWHRyYWluID0gWHRyYWluLCB5dHJhaW4gPSB5dHJhaW4sIAogICAgICAgICAgICAgICAgICAgICAgICAgIFh0ZXN0ID0gWHZhbGlkLCB5dGVzdCA9IHl2YWxpZCwgCiAgICAgICAgICAgICAgICAgICAgICAgICAgbW9kZWxfbGlzdCA9IG1vZGVsX2xpc3QsIHRyX2NvbnRyb2wgPSB0cl9jb250cm9sLCAKICAgICAgICAgICAgICAgICAgICAgICAgICByZXNwb25zZV90eXBlID0gcmVzcG9uc2VfdHlwZSkKY2F0KHNwcmludGYoIkZpdHRpbmcgdGltZSB0YWtlbjogJWYgbWludXRlcyIsIAogICAgICAgICAgICBkaWZmdGltZShTeXMudGltZSgpLCBzdGFydF90aW1lLCB1bml0cyA9ICJtaW5zIikpKQpgYGAKCmBgYHtyIHRpZHltb2RlbHMtZml0cywgZXZhbCA9IHBhcmFtcyRtb2RlbGluZ19wa2cgPT0gInRpZHltb2RlbHMiLCBlY2hvID0gcGFyYW1zJG1vZGVsaW5nX3BrZyA9PSAidGlkeW1vZGVscyJ9CiMgVE9ETzogYWRkIGNvZGUgZm9yIHR1bmluZyBwYXJhbWV0ZXJzCiMgQUs6IE9uY2UgdHVuaW5nIGlzIGZpeGVkLCBjYW4gYWRqdXN0IHNwbGl0cyBmb3Igc3RhYmlsaXR5CgojIyBQaWNrIG1vZGVscyBhbmQgZml0dGluZyBiYWNrZW5kIGZvciB0aWR5bW9kZWxzCgojIGFzIGlzLCBvbmx5IHdvcmtzIGZvciBjbGFzc2lmaWNhdGlvbgpyZl9tb2RlbCA8LSBwYXJzbmlwOjpyYW5kX2ZvcmVzdCgpICU+JQogIHBhcnNuaXA6OnNldF9hcmdzKG10cnkgPSB0dW5lOjp0dW5lKCkpICU+JQogIHBhcnNuaXA6OnNldF9lbmdpbmUoInJhbmdlciIsIGltcG9ydGFuY2UgPSAiaW1wdXJpdHkiKSAlPiUKICBwYXJzbmlwOjpzZXRfbW9kZSgiY2xhc3NpZmljYXRpb24iKQpyZl9ncmlkIDwtIHRpZHlyOjpjcm9zc2luZyhtdHJ5ID0gMTo0KQoKc3ZtX21vZGVsIDwtIHBhcnNuaXA6OnN2bV9yYmYoKSAlPiUKICBwYXJzbmlwOjpzZXRfZW5naW5lKCJrZXJubGFiIikgJT4lCiAgcGFyc25pcDo6c2V0X21vZGUoImNsYXNzaWZpY2F0aW9uIikKCmtubl9tb2RlbCA8LSBwYXJzbmlwOjpuZWFyZXN0X25laWdoYm9yKCkgJT4lCiAgIHBhcnNuaXA6OnNldF9hcmdzKG5laWdoYm9ycyA9IHR1bmUoKSwgd2VpZ2h0X2Z1bmMgPSB0dW5lKCkpICU+JSAKICAgcGFyc25pcDo6c2V0X2VuZ2luZSgia2tubiIpICU+JSAKICAgcGFyc25pcDo6c2V0X21vZGUoImNsYXNzaWZpY2F0aW9uIikKCm1vZGVsX2xpc3QgPC0gbGlzdChSRiA9IGxpc3QobW9kZWwgPSByZl9tb2RlbCwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICBncmlkID0gcmZfZ3JpZCksIAogICAgICAgICAgICAgICAgICAgU1ZNID0gbGlzdChtb2RlbCA9IHN2bV9tb2RlbCwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZ3JpZCA9IE5VTEwpLCAKICAgICAgICAgICAgICAgICAgIEtOTiA9IGxpc3QobW9kZWwgPSBrbm5fbW9kZWwsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGdyaWQgPSA0KSkKa2ZvbGRzIDwtIDUKCnN0YXJ0X3RpbWUgPC0gU3lzLnRpbWUoKQptb2RlbF9yZXN1bHRzIDwtIGZpdFRpZHlNb2RlbHMoWHRyYWluID0gWHRyYWluLCB5dHJhaW4gPSB5dHJhaW4sIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgWHRlc3QgPSBYdmFsaWQsIHl0ZXN0ID0geXZhbGlkLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbW9kZWxfbGlzdCA9IG1vZGVsX2xpc3QsIGtmb2xkcyA9IGtmb2xkcykKY2F0KHNwcmludGYoIkZpdHRpbmcgdGltZSB0YWtlbjogJWYgbWludXRlcyIsIAogICAgICAgICAgICBkaWZmdGltZShTeXMudGltZSgpLCBzdGFydF90aW1lLCB1bml0cyA9ICJtaW5zIikpKQpgYGAKCmBgYHtyIGgyby1maXRzLCBldmFsID0gcGFyYW1zJG1vZGVsaW5nX3BrZyA9PSAiaDJvIiwgZWNobyA9IHBhcmFtcyRtb2RlbGluZ19wa2cgPT0gImgybyJ9CiMjIFBpY2sgbW9kZWxzIGFuZCBmaXR0aW5nIGJhY2tlbmQgZm9yIGgybwptb2RlbF9saXN0IDwtIGxpc3QocmFuZG9tRm9yZXN0ID0gbGlzdChudHJlZXMgPSA1MDApLCAKICAgICAgICAgICAgICAgICAgIHhnYm9vc3QgPSBsaXN0KCkpCgpzdGFydF90aW1lIDwtIFN5cy50aW1lKCkKbW9kZWxfcmVzdWx0cyA8LSBmaXRoMm8oWHRyYWluID0gWHRyYWluLCB5dHJhaW4gPSB5dHJhaW4sCiAgICAgICAgICAgICAgICAgICAgICAgIFh0ZXN0ID0gWHZhbGlkLCB5dGVzdCA9IHl2YWxpZCwKICAgICAgICAgICAgICAgICAgICAgICAgbW9kZWxfbGlzdCA9IG1vZGVsX2xpc3QpCmNhdChzcHJpbnRmKCJGaXR0aW5nIHRpbWUgdGFrZW46ICVmIG1pbnV0ZXMiLCAKICAgICAgICAgICAgZGlmZnRpbWUoU3lzLnRpbWUoKSwgc3RhcnRfdGltZSwgdW5pdHMgPSAibWlucyIpKSkKYGBgCgpgYGB7ciBwcmVkaWN0aW9uLWFjY3VyYWN5LWh0bWwsIHJlc3VsdHMgPSAiYXNpcyIsIGV2YWwgPSBrbml0cjo6aXNfaHRtbF9vdXRwdXQoKX0KIyAoaHRtbCkgdGFibGUgb2YgYWNjdXJhY3kgbWV0cmljcwptb2RlbF9yZXN1bHRzJGVycm9ycyAlPiUKICBzaW1DaGVmOjpwcmV0dHlfRFQoZGlnaXRzID0gMiwgc2lnZmlnID0gRiwgcm93bmFtZXMgPSBGQUxTRSwgCiAgICAgICAgICAgY2FwdGlvbiA9ICJWYWxpZGF0aW9uIFByZWRpY3Rpb24gQWNjdXJhY2llcyIsIAogICAgICAgICAgIG9wdGlvbnMgPSBsaXN0KGRvbSA9ICd0JykpCmBgYAoKYGBge3IgcHJlZGljdGlvbi1hY2N1cmFjeS1sYXRleCwgcmVzdWx0cyA9ICJhc2lzIiwgZXZhbCA9IGtuaXRyOjppc19sYXRleF9vdXRwdXQoKX0KIyAobGF0ZXgpIHRhYmxlIG9mIGFjY3VyYWN5IG1ldHJpY3MKbW9kZWxfcmVzdWx0cyRlcnJvcnMgJT4lCiAgc2ltQ2hlZjo6cHJldHR5X2thYmxlKGRpZ2l0cyA9IDIsIHNpZ2ZpZyA9IEYsIHJvdy5uYW1lcyA9IEZBTFNFLCAKICAgICAgICAgICBjYXB0aW9uID0gIlZhbGlkYXRpb24gUHJlZGljdGlvbiBBY2N1cmFjaWVzIiwKICAgICAgICAgICBmb3JtYXQgPSAibGF0ZXgiKQpgYGAKCiMjIFN0YWJpbGl0eSBjaGVjawoKVGFraW5nIHRoZSBwcmVkaWN0aW9uIG1ldGhvZHMgdGhhdCBwYXNzIHRoZSBwcmVkaWN0aW9uIGNoZWNrLCBwZXJmb3JtIHN0YWJpbGl0eSBhbmFseXNpcy4KCi0gICBTcGVjaWZ5IGFuZCBqdXN0aWZ5IHRoZSBhcHByb3ByaWF0ZSBkYXRhIHBlcnR1cmJhdGlvbihzKS4KLSAgIFJlLWZpdCB0aGUgcHJlZGljdGlvbiBtZXRob2RzIG9uIHRoZXNlIHBlcnR1cmJlZCBkYXRhIHNldHMuCi0gICBFdmFsdWF0ZSBwcmVkaWN0aW9uIG1ldGhvZHMgb24gdmFsaWRhdGlvbiBkYXRhLgotICAgQXNzZXNzIHN0YWJpbGl0eSBhY3Jvc3MgdGhlIGRhdGEgcGVydHVyYmF0aW9ucyBhcyB3ZWxsIGFzIGFjcm9zcyB0aGUgdmFyaW91cyBtZXRob2RzLgotICAgRmlsdGVyIG91dCBwb29yIG1vZGVscyB3aGVyZSBuZWNlc3NhcnkgYW5kIGludGVycHJldCBzdGFiaWxpdHkgcmVzdWx0cy4KCmBgYHthc2lzLCBpbnRlcmFjdGl2ZV90ZXh0ID0gVFJVRX0KCmBgYAoKVE9ETzogQWRkIHJlc3VsdHMgZm9yIGgyMCwgdHVuaW5nIG5lZWRzIHRvIGJlIGZpeGVkIGZvciBoMjAgKyB0aWR5bW9kZWxzIDwhLS0gVE9ETzogQW5hIC0gUHJvdmlkZSBzb21lIGV4YW1wbGUgY29kZSBoZXJlIChmb3IgYm90aCBmaXR0aW5nIGFuZCB2aXN1YWxpemluZyByZXN1bHRzKSBzbyB0aGF0IHRoZSBwcmFjdGl0aW9uZXIgY2FuIGVhc2lseSBpbnB1dCB0aGVpciBkYXRhIGFuZCBtb2RlbHMuIEFkZCBzb21ldGhpbmcgbGlrZSBKYW1lcycgc2xpZGUuIEEgZmV3IHBpY3R1cmVzIHdpdGggcG9zc2libGUgZGF0YSBwZXJ0dXJiYXRpb24gc2NoZW1lcyAoc2VwYXJhdGUgZnJvbSBwYXJhbWV0ZXIgdHVuaW5nKS4gRGF0YSBzcGxpdHRpbmcgdnMgc2FtcGxpbmcgb2Ygb2JzZXJ2YXRpb25zLiBDcm9zcy12YWxpZGF0aW9uLWlzaCBzY2hlbWUgYW5kIGZpeGVkIHRyYWluaW5nL3ZhbGlkYXRpb24gc2V0dXAgKGJvb3RzdHJhcHBpbmcsIHN1Yi1zYW1wbGluZywgc3RyYXRpZmllZC1zYW1wbGluZykuIEFkZCBwYXJhbWV0ZXIgdG8gaW5jbHVkZS9leGNsdWRlIGNlcnRhaW4gY29kZSBjaHVua3MuIC0tPgoKYGBge3Igc3RhYmlsaXR5LWNoZWNrfQpuX3JlcHMgPC0gMiAjIGluY3JlYXNlIGZvciBiZXR0ZXIgc3RhYmlsaXR5IG1lYXN1cmVzIHdoZW4gbm90IHRlc3RpbmcgY29kZQoKIyBwIDwtIHByb2dyZXNzcjo6cHJvZ3Jlc3NvcihzdGVwcyA9IG5fcmVwcykKIyBmdXR1cmU6OnBsYW4obXVsdGlzZXNzaW9uLCB3b3JrZXJzID0gbWluKG5fcmVwcywgcGFyYWxsZWw6OmRldGVjdENvcmVzKCkgLSAxKSkKYm9vdHN0cmFwX21vZGVsX3Jlc3VsdHMgPC0gZnV0dXJlLmFwcGx5OjpmdXR1cmVfcmVwbGljYXRlKAogIG4gPSBuX3JlcHMsCiAgZXhwciA9IHsKICAgIGJvb3RzdHJhcCA8LSBzYW1wbGUoMTpucm93KFh0cmFpbiksIG5yb3coWHRyYWluKSwgcmVwbGFjZSA9IFRSVUUpCiAgICBYdHJhaW5fYiA8LSBYdHJhaW5bYm9vdHN0cmFwLCBdCiAgICB5dHJhaW5fYiA8LSB5dHJhaW5bYm9vdHN0cmFwXQogICAgCiAgICBpZiAocGFyYW1zJG1vZGVsaW5nX3BrZyA9PSAiY2FyZXQiKSB7CiAgICAgIG1vZGVsX3Jlc3VsdHNfYiA8LSBmaXRDYXJldChYdHJhaW4gPSBYdHJhaW5fYiwgeXRyYWluID0geXRyYWluX2IsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgWHRlc3QgPSBYdmFsaWQsIHl0ZXN0ID0geXZhbGlkLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIG1vZGVsX2xpc3QgPSBtb2RlbF9saXN0LCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHRyX2NvbnRyb2wgPSB0cl9jb250cm9sLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHJlc3BvbnNlX3R5cGUgPSByZXNwb25zZV90eXBlKQogICAgfSBlbHNlIGlmIChwYXJhbXMkbW9kZWxpbmdfcGtnID09ICJ0aWR5bW9kZWxzIikgewogICAgICBtb2RlbF9yZXN1bHRzX2IgPC0gZml0VGlkeU1vZGVscyhYdHJhaW4gPSBYdHJhaW5fYiwgeXRyYWluID0geXRyYWluX2IsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBYdGVzdCA9IFh2YWxpZCwgeXRlc3QgPSB5dmFsaWQsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIG1vZGVsX2xpc3QgPSBtb2RlbF9saXN0LCBrZm9sZHMgPSBrZm9sZHMpCiAgICB9IGVsc2UgaWYgKHBhcmFtcyRtb2RlbGluZ19wa2cgPT0gImgybyIpIHsKICAgICAgbW9kZWxfcmVzdWx0c19iIDwtIGZpdGgybyhYdHJhaW4gPSBYdHJhaW5fYiwgeXRyYWluID0geXRyYWluX2IsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgWHRlc3QgPSBYdmFsaWQsIHl0ZXN0ID0geXZhbGlkLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIG1vZGVsX2xpc3QgPSBtb2RlbF9saXN0KQogICAgfQogICAgcmV0dXJuKG1vZGVsX3Jlc3VsdHNfYikKICB9LAogIHNpbXBsaWZ5ID0gRkFMU0UKKQoKYm9vdHN0cmFwX21vZGVsX2VycnMgPC0gcHVycnI6Om1hcF9kZnIoYm9vdHN0cmFwX21vZGVsX3Jlc3VsdHMsICJlcnJvcnMiLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLmlkID0gImJvb3RzdHJhcF9pZCIpCmJvb3RzdHJhcF9tb2RlbF9wcmVkcyA8LSBwdXJycjo6bWFwKGJvb3RzdHJhcF9tb2RlbF9yZXN1bHRzLCAicHJlZGljdGlvbnMiLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAuaWQgPSAiYm9vdHN0cmFwX2lkIikKYm9vdHN0cmFwX21vZGVsX2ltcHMgPC0gcHVycnI6Om1hcChib290c3RyYXBfbW9kZWxfcmVzdWx0cywgImltcG9ydGFuY2UiLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC5pZCA9ICJib290c3RyYXBfaWQiKSAlPiUKICBkcGx5cjo6YmluZF9yb3dzKC5pZCA9ICJib290c3RyYXBfaWQiKQoKYm9vdHN0cmFwX21vZGVsX2VycnNfc3VtbWFyeSA8LSBib290c3RyYXBfbW9kZWxfZXJycyAlPiUKICBkcGx5cjo6Z3JvdXBfYnkobW9kZWwpICU+JQogIGRwbHlyOjpzdW1tYXJpc2UoZHBseXI6OmFjcm9zcygtYm9vdHN0cmFwX2lkLCBsaXN0KG1lYW4gPSBtZWFuLCBzZCA9IHNkKSkpICU+JQogIHNldE5hbWVzKHN0cmluZ3I6OnN0cl9yZXBsYWNlKGNvbG5hbWVzKC4pLCAiXyIsICIgIikpCmBgYAoKYGBge3Igc3RhYmlsaXR5LWNoZWNrLXRhYmxlLWh0bWwsIHJlc3VsdHMgPSAiYXNpcyIsIGV2YWwgPSBrbml0cjo6aXNfaHRtbF9vdXRwdXQoKX0KIyAoaHRtbCkgdGFibGUgb2YgYWNjdXJhY3kgbWV0cmljcwpib290c3RyYXBfbW9kZWxfZXJyc19zdW1tYXJ5ICU+JQogIHNpbUNoZWY6OnByZXR0eV9EVCgKICAgIGRpZ2l0cyA9IDIsIHNpZ2ZpZyA9IEYsIHJvd25hbWVzID0gRkFMU0UsIAogICAgY2FwdGlvbiA9ICJWYWxpZGF0aW9uIFByZWRpY3Rpb24gQWNjdXJhY2llcyBPdmVyIEJvb3RzdHJhcHBlZCBUcmFpbmluZyBGaXRzIiwgCiAgICBib2xkX2Z1bmN0aW9uID0gIi4gPT0gbWF4KC4sIG5hLnJtID0gVFJVRSkiLCBib2xkX21hcmdpbiA9IDIsCiAgICBib2xkX3NjaGVtZSA9IGMoRiwgVCwgRiwgVCwgRiksCiAgICBvcHRpb25zID0gbGlzdChkb20gPSAndCcpCiAgKQpgYGAKCmBgYHtyIHN0YWJpbGl0eS1jaGVjay10YWJsZS1sYXRleCwgcmVzdWx0cyA9ICJhc2lzIiwgZXZhbCA9IGtuaXRyOjppc19sYXRleF9vdXRwdXQoKX0KIyAobGF0ZXgpIHRhYmxlIG9mIGFjY3VyYWN5IG1ldHJpY3MKYm9vdHN0cmFwX21vZGVsX2VycnNfc3VtbWFyeSAlPiUKICBzaW1DaGVmOjpwcmV0dHlfa2FibGUoCiAgICBkaWdpdHMgPSAyLCBzaWdmaWcgPSBGLCByb3cubmFtZXMgPSBGQUxTRSwgCiAgICBjYXB0aW9uID0gIlZhbGlkYXRpb24gUHJlZGljdGlvbiBBY2N1cmFjaWVzIE92ZXIgQm9vdHN0cmFwcGVkIFRyYWluaW5nIEZpdHMiLCAKICAgIGJvbGRfZnVuY3Rpb24gPSAiLiA9PSBtYXgoLiwgbmEucm0gPSBUUlVFKSIsIGJvbGRfbWFyZ2luID0gMiwKICAgIGJvbGRfc2NoZW1lID0gYyhGLCBULCBGLCBULCBGKSwKICAgIGZvcm1hdCA9ICJsYXRleCIKICApCmBgYAoKYGBge3Igc3RhYmlsaXR5LWNoZWNrLWJveHBsb3R9CiMgYm94cGxvdHMKYm9vdHN0cmFwX21vZGVsX2VycnMgJT4lCiAgdGlkeXI6OnBpdm90X2xvbmdlcihjb2xzID0gLWMobW9kZWwsIGJvb3RzdHJhcF9pZCksIAogICAgICAgICAgICAgICAgICAgICAgbmFtZXNfdG8gPSAiTWV0cmljIiwgdmFsdWVzX3RvID0gIlZhbHVlIikgJT4lCiAgcGxvdEJveHBsb3QoeF9zdHIgPSAiVmFsdWUiLCB5X3N0ciA9ICJtb2RlbCIsIGhvcml6b250YWwgPSBGQUxTRSkgKwogIGdncGxvdDI6OmZhY2V0X3dyYXAofiBNZXRyaWMpICsKICBnZ3Bsb3QyOjpsYWJzKHggPSAiTWV0aG9kIiwgCiAgICAgICAgICAgICAgICB0aXRsZSA9ICJWYWxpZGF0aW9uIFByZWRpY3Rpb24gQWNjdXJhY2llcyBPdmVyIEJvb3RzdHJhcHBlZCBUcmFpbmluZyBGaXRzIikKYGBgCgojIyBJbnRlcnByZXRhYmlsaXR5IHsudGFic2V0IC5jdXN0b20tdGFic30KCkZvciB0aGUgbW9kZWxzIHRoYXQgcGFzcyB0aGUgcHJlZGljdGlvbiBhbmQgc3RhYmlsaXR5IGNoZWNrcywgZXh0cmFjdCB0aGUgaW1wb3J0YW50IGZlYXR1cmVzIGluIHRoZSBwcmVkaWN0aXZlIG1vZGVscyB0aGF0IGFyZSBzdGFibGUgYWNyb3NzIGJvdGggZGF0YSBhbmQgbW9kZWwgcGVydHVyYmF0aW9ucy4gRGV0ZXJtaW5pbmcgdGhlIGltcG9ydGFuY2Ugb2YgYSBmZWF0dXJlIGNhbiBiZSBtZXRob2QgZGVwZW5kZW50LgoKYGBge2FzaXMsIGludGVyYWN0aXZlX3RleHQgPSBUUlVFfQoKYGBgCgojIyMgRnVsbCBNb2RlbCAod2l0aG91dCBzdGFiaWxpdHkpIHsudW5udW1iZXJlZCAudGFic2V0IC50YWJzZXQtcGlsbHN9CgojIyMjIFRhYmxlIHsudW5udW1iZXJlZH0KCmBgYHtyIGFkZF9uZXdfbGluZSA9IFRSVUUsIHJlc3VsdHMgPSAiYXNpcyIsIGV2YWwgPSBrbml0cjo6aXNfaHRtbF9vdXRwdXQoKX0Kc2ltQ2hlZjo6cHJldHR5X0RUKG1vZGVsX3Jlc3VsdHMkaW1wb3J0YW5jZSwgZGlnaXRzID0gMiwgc2lnZmlnID0gRiwgCiAgICAgICAgICAgICAgICAgICBjYXB0aW9uID0gIlZhcmlhYmxlIEltcG9ydGFuY2VzIikKYGBgCgpgYGB7ciBhZGRfbmV3X2xpbmUgPSBUUlVFLCByZXN1bHRzID0gImFzaXMiLCBldmFsID0ga25pdHI6OmlzX2xhdGV4X291dHB1dCgpfQpzaW1DaGVmOjpwcmV0dHlfa2FibGUobW9kZWxfcmVzdWx0cyRpbXBvcnRhbmNlLCBkaWdpdHMgPSAyLCBzaWdmaWcgPSBGLCAKICAgICAgICAgICAgICAgICAgICAgIGNhcHRpb24gPSAiVmFyaWFibGUgSW1wb3J0YW5jZXMiLCBmb3JtYXQgPSAibGF0ZXgiKQpgYGAKCiMjIyMgUGxvdHMgey51bm51bWJlcmVkfQoKYGBge3IsIGFkZF9uZXdfbGluZSA9IFRSVUV9CnBsb3RGZWF0dXJlSW1wb3J0YW5jZShtb2RlbF9yZXN1bHRzJGltcG9ydGFuY2UsCiAgICAgICAgICAgICAgICAgICAgICB1c2VfcmFua2luZ3MgPSBGQUxTRSwKICAgICAgICAgICAgICAgICAgICAgIHVzZV9mYWNldHMgPSBUUlVFLAogICAgICAgICAgICAgICAgICAgICAgaW50ZXJhY3RpdmUgPSBGQUxTRSkKYGBgCgpgYGB7ciwgYWRkX25ld19saW5lID0gVFJVRX0KcGxvdEZlYXR1cmVJbXBvcnRhbmNlUGFpcihtb2RlbF9yZXN1bHRzJGltcG9ydGFuY2UsCiAgICAgICAgICAgICAgICAgICAgICAgICAgdXNlX3JhbmtpbmdzID0gVFJVRSwKICAgICAgICAgICAgICAgICAgICAgICAgICBpbnRlcmFjdGl2ZSA9IEZBTFNFKQpgYGAKCiMjIyBCb290c3RyYXBwZWQgTW9kZWwgKHdpdGggc3RhYmlsaXR5KSB7LnVubnVtYmVyZWQgLnRhYnNldCAudGFic2V0LXBpbGxzfQoKIyMjIyBUYWJsZSB7LnVubnVtYmVyZWR9CgpgYGB7ciBhZGRfbmV3X2xpbmUgPSBUUlVFLCByZXN1bHRzID0gImFzaXMiLCBldmFsID0ga25pdHI6OmlzX2h0bWxfb3V0cHV0KCl9CmJvb3RzdHJhcF9tb2RlbF9pbXBzX3N1bW1hcnkgPC0gYm9vdHN0cmFwX21vZGVsX2ltcHMgJT4lCiAgZHBseXI6Omdyb3VwX2J5KG1vZGVsLCB2YXJpYWJsZSkgJT4lCiAgZHBseXI6OnN1bW1hcmlzZShgTWVhbiBJbXBvcnRhbmNlYCA9IG1lYW4oaW1wb3J0YW5jZSksCiAgICAgICAgICAgICAgICAgICBgTWVkaWFuIEltcG9ydGFuY2VgID0gbWVkaWFuKGltcG9ydGFuY2UpLAogICAgICAgICAgICAgICAgICAgYFNEIEltcG9ydGFuY2VgID0gc2QoaW1wb3J0YW5jZSksCiAgICAgICAgICAgICAgICAgICBgTWluIEltcG9ydGFuY2VgID0gbWluKGltcG9ydGFuY2UpLAogICAgICAgICAgICAgICAgICAgYE1heCBJbXBvcnRhbmNlYCA9IG1heChpbXBvcnRhbmNlKSkKc2ltQ2hlZjo6cHJldHR5X0RUKGJvb3RzdHJhcF9tb2RlbF9pbXBzX3N1bW1hcnksIAogICAgICAgICAgICAgICAgICAgZGlnaXRzID0gMiwgc2lnZmlnID0gRiwgCiAgICAgICAgICAgICAgICAgICBjYXB0aW9uID0gIlN1bW1hcnkgb2YgdmFyaWFibGUgaW1wb3J0YW5jZXMgYWNyb3NzIGJvb3RzdHJhcHBlZCBtb2RlbHMiKQpgYGAKCmBgYHtyIGFkZF9uZXdfbGluZSA9IFRSVUUsIHJlc3VsdHMgPSAiYXNpcyIsIGV2YWwgPSBrbml0cjo6aXNfbGF0ZXhfb3V0cHV0KCl9CmJvb3RzdHJhcF9tb2RlbF9pbXBzX3N1bW1hcnkgPC0gYm9vdHN0cmFwX21vZGVsX2ltcHMgJT4lCiAgZHBseXI6Omdyb3VwX2J5KG1vZGVsLCB2YXJpYWJsZSkgJT4lCiAgZHBseXI6OnN1bW1hcmlzZShgTWVhbiBJbXBvcnRhbmNlYCA9IG1lYW4oaW1wb3J0YW5jZSksCiAgICAgICAgICAgICAgICAgICBgTWVkaWFuIEltcG9ydGFuY2VgID0gbWVkaWFuKGltcG9ydGFuY2UpLAogICAgICAgICAgICAgICAgICAgYFNEIEltcG9ydGFuY2VgID0gc2QoaW1wb3J0YW5jZSksCiAgICAgICAgICAgICAgICAgICBgTWluIEltcG9ydGFuY2VgID0gbWluKGltcG9ydGFuY2UpLAogICAgICAgICAgICAgICAgICAgYE1heCBJbXBvcnRhbmNlYCA9IG1heChpbXBvcnRhbmNlKSkKc2ltQ2hlZjo6cHJldHR5X2thYmxlKGJvb3RzdHJhcF9tb2RlbF9pbXBzX3N1bW1hcnksIAogICAgICAgICAgICAgICAgICAgICAgZGlnaXRzID0gMiwgc2lnZmlnID0gRiwgCiAgICAgICAgICAgICAgICAgICAgICBjYXB0aW9uID0gIlN1bW1hcnkgb2YgdmFyaWFibGUgaW1wb3J0YW5jZXMgYWNyb3NzIGJvb3RzdHJhcHBlZCBtb2RlbHMiLAogICAgICAgICAgICAgICAgICAgICAgZm9ybWF0ID0gImxhdGV4IikKYGBgCgojIyMjIFBsb3RzIHsudW5udW1iZXJlZH0KCmBgYHtyLCBhZGRfbmV3X2xpbmUgPSBUUlVFfQpwbG90RmVhdHVyZUltcG9ydGFuY2VTdGFiaWxpdHkoYm9vdHN0cmFwX21vZGVsX2ltcHMsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB1c2VfcmFua2luZ3MgPSBGQUxTRSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHVzZV9mYWNldHMgPSBGQUxTRSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGludGVyYWN0aXZlID0gRkFMU0UpCmBgYAoKIyBNYWluIFJlc3VsdHMKCkludGVycHJldCBhbmQgc3VtbWFyaXplIHRoZSBwcmVkaWN0aW9uIGFuZCBzdGFiaWxpdHkgcmVzdWx0cy4KCmBgYHthc2lzLCBpbnRlcmFjdGl2ZV90ZXh0ID0gVFJVRX0KCmBgYAoKRXZhbHVhdGUgcGlwZWxpbmUgb24gdGVzdCBkYXRhLgoKYGBge3IgZmluYWwtZml0cywgZXZhbCA9IFRSVUUgfQpYdHJhaW5fZmluYWwgPC0gZHBseXI6OmJpbmRfcm93cyhYdHJhaW4sIFh2YWxpZCkKeXRyYWluX2ZpbmFsIDwtIGMoeXRyYWluLCB5dmFsaWQpCmlmIChwYXJhbXMkbW9kZWxpbmdfcGtnID09ICJjYXJldCIpIHsKICBmaW5hbF9tb2RlbF9yZXN1bHRzIDwtIGZpdENhcmV0KFh0cmFpbiA9IFh0cmFpbl9maW5hbCwgeXRyYWluID0geXRyYWluX2ZpbmFsLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFh0ZXN0ID0gWHRlc3QsIHl0ZXN0ID0geXRlc3QsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbW9kZWxfbGlzdCA9IG1vZGVsX2xpc3QsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdHJfY29udHJvbCA9IHRyX2NvbnRyb2wsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgcmVzcG9uc2VfdHlwZSA9IHJlc3BvbnNlX3R5cGUpCn0gZWxzZSBpZiAocGFyYW1zJG1vZGVsaW5nX3BrZyA9PSAidGlkeW1vZGVscyIpIHsKICBmaW5hbF9tb2RlbF9yZXN1bHRzIDwtIGZpdFRpZHlNb2RlbHMoWHRyYWluID0gWHRyYWluX2ZpbmFsLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgeXRyYWluID0geXRyYWluX2ZpbmFsLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgWHRlc3QgPSBYdGVzdCwgeXRlc3QgPSB5dGVzdCwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbW9kZWxfbGlzdCA9IG1vZGVsX2xpc3QsIGtmb2xkcyA9IGtmb2xkcykKfSBlbHNlIGlmIChwYXJhbXMkbW9kZWxpbmdfcGtnID09ICJoMm8iKSB7CiAgZmluYWxfbW9kZWxfcmVzdWx0cyA8LSBmaXRoMm8oWHRyYWluID0gWHRyYWluX2ZpbmFsLCB5dHJhaW4gPSB5dHJhaW5fZmluYWwsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgWHRlc3QgPSBYdGVzdCwgeXRlc3QgPSB5dGVzdCwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBtb2RlbF9saXN0ID0gbW9kZWxfbGlzdCkKfQoKYGBgCgpTdW1tYXJpemUgdGVzdCBzZXQgcHJlZGljdGlvbiBhbmQvb3IgaW50ZXJwcmV0YWJpbGl0eSByZXN1bHRzLgoKYGBge2FzaXMsIGludGVyYWN0aXZlX3RleHQgPSBUUlVFfQoKYGBgCgo8IS0tIFRPRE86IEFuYSAtIGFkZCB0ZW1wbGF0ZSB0YWJsZXMgd2l0aCBpbnRlcnByZXRhdGlvbiAtLT4KCmBgYHtyIHRlc3QtYWNjdXJhY3ktdGFibGUtaHRtbCwgcmVzdWx0cyA9ICJhc2lzIiwgZXZhbCA9IGtuaXRyOjppc19odG1sX291dHB1dCgpfQojIChodG1sKSB0YWJsZSBvZiBhY2N1cmFjeSBtZXRyaWNzCmZpbmFsX21vZGVsX3Jlc3VsdHMkZXJyb3JzICU+JQogIHNpbUNoZWY6OnByZXR0eV9EVChkaWdpdHMgPSAyLCBzaWdmaWcgPSBGLCByb3duYW1lcyA9IEZBTFNFLCAKICAgICAgICAgICBjYXB0aW9uID0gIlRlc3QgUHJlZGljdGlvbiBBY2N1cmFjaWVzIiwgCiAgICAgICAgICAgb3B0aW9ucyA9IGxpc3QoZG9tID0gJ3QnKSkKYGBgCgpgYGB7ciB0ZXN0LWFjY3VyYWN5LXRhYmxlLWxhdGV4LCByZXN1bHRzID0gImFzaXMiLCBldmFsID0ga25pdHI6OmlzX2xhdGV4X291dHB1dCgpfQojIChsYXRleCkgdGFibGUgb2YgYWNjdXJhY3kgbWV0cmljcwpmaW5hbF9tb2RlbF9yZXN1bHRzJGVycm9ycyAlPiUKICAgIHNpbUNoZWY6OnByZXR0eV9rYWJsZShkaWdpdHMgPSAyLCBzaWdmaWcgPSBGLCByb3cubmFtZXMgPSBGQUxTRSwgCiAgICAgICAgICAgICBjYXB0aW9uID0gIlRlc3QgUHJlZGljdGlvbiBBY2N1cmFjaWVzIiwgZm9ybWF0ID0gImxhdGV4IikKYGBgCgojIFBvc3QgaG9jIGFuYWx5c2lzCgpNb3ZlIGJleW9uZCB0aGUgZ2xvYmFsIHByZWRpY3Rpb24gYWNjdXJhY3kgbWV0cmljcyBhbmQgZGl2ZSBkZWVwZXIgaW50byBpbmRpdmlkdWFsLWxldmVsIHByZWRpY3Rpb25zIGZvciB0aGUgdmFsaWRhdGlvbiBhbmQvb3IgdGVzdCBzZXQsIGkuZS4sIHByb3ZpZGUgYSBtb3JlICJsb2NhbCIgYW5hbHlzaXMuCgotICAgRXhhbWluZSBhbnkgcG9pbnRzIHRoYXQgaGFkIHBvb3IgcHJlZGljdGlvbnMuCi0gICBFeGFtaW5lIGRpZmZlcmVuY2VzIGJldHdlZW4gcHJlZGljdGlvbiBtZXRob2RzLgoKYGBge2FzaXMsIGludGVyYWN0aXZlX3RleHQgPSBUUlVFfQoKYGBgCgpgYGB7cn0KcGxvdFBhaXJzKGNiaW5kKGBUcnVlIFJlcHNvbnNlc2AgPSB5dGVzdCwgZmluYWxfbW9kZWxfcmVzdWx0cyRwcmVkaWN0aW9ucyksIAogICAgICAgICAgY29sdW1ucyA9IDE6KG5jb2woZmluYWxfbW9kZWxfcmVzdWx0cyRwcmVkaWN0aW9ucykgKyAxKSwgCiAgICAgICAgICB0aXRsZSA9ICJDb21wYXJpc29uIG9mIG1vZGVsIHByZWRpY3Rpb25zIikKYGBgCgojIENvbmNsdXNpb25zCgpSZWl0ZXJhdGUgbWFpbiBmaW5kaW5ncywgbm90ZSBhbnkgY2F2ZWF0cywgYW5kIGNsZWFybHkgdHJhbnNsYXRlIGZpbmRpbmdzL2FuYWx5c2lzIGJhY2sgdG8gdGhlIGRvbWFpbiBwcm9ibGVtIGNvbnRleHQuCgpgYGB7YXNpcywgaW50ZXJhY3RpdmVfdGV4dCA9IFRSVUV9CgpgYGAKCg==

PCS Documentation

December 13, 2021

1 Domain problem formulation

2 Data

2.1 Data Collection

2.2 Data Splitting

2.2.1 Data Splitting Overview

X Data Split

Y Data Split

2.3 Data Cleaning and Preprocessing

2.4 Data Exploration

2.4.1 Training Data Overview

Summary Tables

X distribution

Y distribution

Feature Correlation

Feature Pair Plots

Marginal Associations

PCA

3 Prediction Modeling

3.1 Prediction check

3.2 Stability check

3.3 Interpretability

Full Model (without stability)

Table

Plots

Bootstrapped Model (with stability)

Table

Plots

4 Main Results

5 Post hoc analysis

6 Conclusions